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Abstract 

We consider the problem of reliable communication over non-binary insertion/deletion channels where 
symbols are randomly deleted from or inserted in the transmitted sequence and all symbols are corrupted 
by additive white Gaussian noise. To this end, we utilize the inherent redundancy achievable in non-binary 
symbol sets by first expanding the symbol set and then allocating part of the bits associated with each 
symbol to watermark symbols. The watermark sequence, known at the receiver, is then used by a forward- 
backward algorithm to provide soft information for an outer code which decodes the transmitted sequence. 
Through numerical results and discussions, we evaluate the performance of the proposed solution and 
show that it leads to significant system ability to detect and correct insertios/deletions. We also provide 
estimates of the maximum achievable information rates of the system, compare them with the available 
bounds, and construct practical codes capable of approaching these limits. 

Index Terms 

concatenated coding, error-correction coding, insertion/deletion channels, watermark codes, hidden 
Markov models (HMM) 



I. Introduction 

Since the seminal work of Shannon |[T1, there have been huge advancements in coding and information 
theory. The fundamental limits and efficient coding solutions approaching these limits are now known for 
many communication channels. However, in the vast majority of coding schemes invented, it is assumed 
that the receiver is perfectly synchronized with the transmitter, i.e., the symbol arrival times are known at 
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the receiver. In most communication systems, however, achieving perfect synchronization is not possible 
even with the existence of timing recovery systems. 

When perfect synchronization does not exist, random symbol insertions and deletions (synchronization 
errors) occur in the received sequence. This phenomenon poses a great challenge for error correction. 
Since the positions of the inserted/deleted symbols are unknown at the receiver, even a single uncorrected 
insertion/deletion can result in a catastrophic burst of errors. Thus, conventional error-correcting codes fail 
at these situations. 

Error-correcting codes designed for dealing with such insertion/deletion (I/D) channels are called syn- 
chronization codes. Synchronization codes have a long history but their design and analysis has proven to 
be extremely challenging, hence few practical results exist in the literature. Moreover, standard approaches 
do not lead to finding the optimal codebooks or tight bounds on the capacity of I/D channels and finding 
their capacity is still an open problem |]2]. 

The first synchronization code was proposed by Sellers in 1962 |[3l. He inserted marker sequences in 
the transmitted bitstream to achieve synchronization. Long markers allowed the decoder to correct multiple 
insertion or deletion errors but greatly increased the overhead. In 1966, using number-theoretic techniques, 
Levenshtein constructed binary codes capable of correcting a single insertion or deletion assuming that 
the codeword boundaries were known at the decoder |@]. Most subsequent work were inspired by the 
number-theoretic methods used by Levenshtein, e.g., see im-lH. Unfortunately, these constructions either 
cannot be generalized to correct multiple synchronization errors without a significant loss in rate, do not 
scale well for large block lengths, or lack practical and efficient encoding or decoding algorithms. 

Some authors also generalized these number-theoretic methods to non-binary alphabets and constructed 
non-binary synchronization codes ll9l- |[T3l . Following llT2l . perfect deletion-correcting codes were studied 
and constructed using combinatorial approaches llT4l - l|T6l . Most of these codes, however, are constructed 
using ad hoc techniques and no practical encoding and decoding algorithm is provided. Non-binary low- 
density parity-check (LDPC) codes decoded by a verification-based decoding algorithm are designed for 
deletion channels in ifTTl . Unfortunately, the decoding complexity of this construction is also far from 
being practical. 

The drawback of all the above-mentioned synchronization codes is that they only work under very 
stringent synchronization and noise restrictions such as working only on deletion channels, or a single 
synchronization error per block. Coding methods proposed for error-correction on the I/D channels working 
under more general conditions are usually based on concatenated coding schemes with two layers of codes, 
i.e., an inner and an outer code ||T81 - ||22| . The inner code identifies the positions of the synchronization 
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errors and the outer code is responsible for correcting the insertions, deletions, and substitution errors as 
well as misidentified synchronization errors. 

In the seminal work of Davey and MacKay [T9ll, a practical concatenated coding method is presented 
for error-correction on general binary I/D channels. They have called their inner code, a watermark code. 
The main idea is to provide a carrier signal or watermark for the outer code. The synchronization errors 
are inferred by the outer code via identifying discontinuities in the carrier signal. One of the advantages of 
watermark codes is that the decoder does not need to know the block boundaries of the received sequence. 
However, due to the use of a sparsifier, rate loss is significant. The watermark is substituted by fixed and 
pseudo-random markers in ED and is shown that it allows better rates but is only able to outperform the 
watermark codes at low synchronization error rates. Also, it has recently been shown that the performance 
of watermark codes can be improved by using symbol-level decoding instead of bit-level decoding 123)1 . 

In this work, we consider the problem of devising an efficient coding method for reliable communication 
over non-binary I/D channels. On these channels, synchronization errors occur at the symbol level, i.e., 
symbols are randomly inserted in and deleted from the received sequence. We also assume that all symbols 
are corrupted by additive white Gaussian noise (AWGN). The use of this channel model is motivated by 
the fact that at the receiver the received continuous waveform is first sampled at certain time instances 
to produce the discrete symbol sequence required by the decoder If the symbol arrival times are not 
perfectly known at the receiver, i.e., there is timing mismatch, some of the transmitted symbols are not 
sampled at all (symbol deletions) or sampled multiple times (symbol insertions) ll25l . As a result, this 
channel model can be used to represent non-binary communications over the AWGN channel suffering 
from timing mismatch. Most communication systems use non-binary signalling, where synchronization 
errors can result in insertion/deletion at the symbol level. 

For the proposed channel model, we utilize the inherent redundancy that can be achieved in non-binary 
symbol sets by first expanding the symbol set and then allocating part of the bits associated with each 
symbol to watermark symbols. As a result, not all the available bits in the signal constellation are used for 
the transmission of information bits. In its simplest form, our solution can be viewed as a communication 
system using two different signal sets. The system switches between these two signal sets according to a 
binary watermark sequence. Since the watermark sequence is known both at the transmitter and the receiver, 
probabilistic decoding can be used to infer the insertions and deletions that occurred and to remove the 
effect of additive noise. In particular, the system is modeled by a hidden Markov model (HMM) |[26ll and 
the forward-backward algorithm ||27l is used for decoding. 
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Our proposed scheme resembles trellis coded modulation (TCM) ll28l . The main idea in both methods 
is to add redundancy by expanding the symbol set and limit the symbol transitions in a controlled manner. 
The proposed method is also closely related to the watermark codes of |[T9l . In both methods, decoding is 
done by the aid of a watermark sequence which both the transmitter and receiver agree on. The difference 
is that the extra degree of freedom in non-binary sets allows us to separate information from the watermark. 

Our proposed solution leads to significant system ability to detect and correct synchronization errors. 
For example, a rate 1/4 binary outer code is capable of correcting about 2,900 insertion/deletion errors 
per block of 10, 012 symbols even when block boundaries are unknown at the receiver. 

This paper is organized as follows. In Sections JI] and |lIIJ we state our proposed approach and describe the 
system model. Section JV] demonstrates the capabilities of the proposed solution by providing numerical 
results and discussions. Section |V] describes ways to increase the achievable information rates on the 
channel and Section |Vl] analyzes the system in terms of complexity, and practical considerations. Finally, 
Section IVIII concludes the paper. 

II. Channel model and the proposed approach 

Throughout this paper, scalar quantities are shown by lower case symbols, complex quantities by boldface 
letters, and vectors by underlined symbols. 

A. Channel model 

The channel model we consider in this work is a non-binary I/D channel with AWGN where insertions 
and deletions occur at the symbol level. Similar to |jT9], it is assumed that the symbols from the input 
sequence x first enter a queue before being transmitted. Then at each channel use, either a random symbol 
is inserted in the symbol sequence gc^ with probability p\, the next queued symbol is deleted with probability 
Pd> or the next queued symbol is transmitted (put as the next symbol in x^) with probability pt = 1— pd— Pi- 
For computational purposes, we assume that the maximum number of insertions which can occur at each 
channel use is /. The resulting symbol sequence is finally affected by an i.i.d. sequence of AWGN z_ 
where z ~ CA/'(0, 2cr^) and y = + zis received at the receiver side. 

Note that in this paper, to show the capabilities of the proposed method, we consider totally ran- 
dom and independent symbol insertions/deletions. When symbol insertions are resulted from imperfect 
synchronization, insertions or deletions tend to be correlated. These cases lead to easier identification of 
insertions/deletions at the receiver compared to random independent insertions/deletions which we consider 
here. 
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B. Proposed approach 

Now, consider a communication system working on this channel by employing an M-ary signalling 
(e.g., M-ary phase-shift keying (PSK)). We call this the base system. Motivated by the idea of watermark 
codes |[T9l . we are interested in embedding a watermark in the transmitted sequence. The watermark, 
being known at the receiver, allows the decoder to deduce the insertions and deletions and to recover the 
transmitted sequence. 

The watermark can be embedded in the transmitted sequence in many ways. One way of doing this is 
to add the watermark to the information sequence and treat the information sequence as additive noise at 
the receiver. This is a direct extension of the binary watermark codes of ^19] to non-binary signalling. In 
particular, the additive watermark w can be defined as a sequence of M-ary symbols drawn from the base 
system constellation. The binary information sequence is first passed through a sparsifier; every k bits of 
the information sequence is converted to an n-tuple of M-ary symbols. The rate of the sparsifier is then 
given hy Ts = k/n where < < m and M = 2"^. The average density of the sparsifier / is defined as 
the average Hamming distance of the n-tuples divided by n. The mapping used in the sparsifier is chosen 
as to minimize /. 

By defining addition as shifting over the constellation symbols, the watermark sequence could be added 
to the sparsified messages (denote it by s) and w ® s be sent over the channel. At the receiver, similar 
to |[T9l . an inner decoder which knows the watermark sequence, uses the received sequence to deduce the 
insertions/deletions and provides soft information for an outer code. 

The main drawback of this method is that the decoder is not able to distinguish between additive noise 
and the information symbols. This is because the information is embedded into the watermark by adding 
s to w. Sequence s contains both zeros and non-zero symbols. Non-zero symbols shift the watermark 
symbols over the constellation, similar to what additive noise does. This greatly degrades the performance 
of the decoder. To improve the decoding performance, s should contain as many zeros as possible, i.e., 
be as sparse as possible, which is equivalent to having a small /. A small / is achieved by decreasing rg 
which in turn decreases the achievable rates on the channel directly. Also, notice that even in the absence 
of additive noise, the decoder is still fooled by the shifts occurred over w and thus misidentifies some of 
the insertions/deletions. 

To aid error recovery at the receiver, we are interested in an embedding method which makes the 
watermark as distinguishable as possible from the information sequence. This necessitates using some 
extra resources (other than those used to transmit the information sequence) for transmitting the watermark 
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sequence. These extra resources can be provided by enlarging the signal set. The extra available bits per 
transmission can then be used to transmit the watermark. After embedding the watermark, we refer to the 
system as the watermarked system. 

In this work, we are mostly interested in binary watermark sequences. As a result, to accommodate the 
watermark bits in each symbol, we expand the signal set size M by the factor of 2, giving rise to a 2A/-ary 
signalling scheme. For example, if the base system uses 4-PSK, in the watermarked system we use 8-PSK 
modulation. To provide fair comparison, we put the symbol rate, information bits per symbol (denoted by 
Tc), and average energy of the signal constellation of the watermarked system equal to those of the base 
system. As a result, the spectral efficiency and the total transmitted power of the watermarked system are 
equal to those of the base system. In other words, no bit rate, bandwidth, or power is sacrificed as a result 
of embedding the watermark. 

Notice that Vc = m where M = 2^ for the base system and also the watermarked system when a binary 
watermark sequence is used for each transmitted symbol. This is because in an M-ary base system all the 
m available bits are dedicated to information bits. Also, in each symbol of the 2M-ary watermarked system 
(with m + 1 available bits) m bits are assigned to information bits. Later, we will see that sometimes it is 
more efficient to use non-binary watermark sequences or to assign less than one bit per symbol on average 
to the watermark giving rise to < rc < m + 1. These cases will be investigated in Section Ivl 

Expanding the signal set while fixing the average energy of the constellation leads to reduction in the 
minimum distance of the constellation. Nevertheless, we show that by using the mapping described in 
Section IIII-A[ the minimum distance dmin between symbols corresponding to the same watermark value 
does not necessarily reduce. In fact in some cases, e.g., in PSK modulation, the minimum distance does 
not change compared to the base system. Thus, the noise immunitjQ of the system does not change after 
adding the watermark. 

III. System model 

The proposed system model is shown in Fig. [T] First, the binary information sequence b is encoded by 
the outer code producing the coded binary sequence d which is then broken into m-bit subsequences. The 
modulator then combines the binary watermark w and the m-bit subsequences by a one-to-one mapping 
fi : {0, Ij^+i — > X where X is the signal set of size \X\ = 2M = 2™+^. Then x is sent over the channel. 

'Here, the noise immunity is measured in tlie absence of synclironization eiTors under tlie assumption of minimum distance 
decoding. As a result, tfie minimum distance between tfie signal constellation points can be used as the noise immunity measure. 
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Fig. 1. The proposed system model. 



The received sequence y is first decoded by the watermark decoder which provides soft information for 
the outer decoder in terms of log-likelihood ratios (LLRs). The LLR sequence [ is then utilized to decode 
the information sequence b. 

A. Modulator 

The modulator plays a key role in the proposed system. It allows embedding encoded data and watermark 
bits while ensuring a good minimum distance. The most important part of designing the modulator is to 
choose an appropriate mapping fi. By viewing fj, as {0, 1} x {0, 1}"^ X, we first divide X into two 
disjoint subsets X^ and X^ each having M signal points corresponding to watermark bit it; = and w = \, 
respectively. Thus, in the label of each signal point, one bit (can be any of the m + 1 bits) is dedicated to 
the watermark bit and the other m bits correspond to the m-bit subsequences of d. Formally we have 

X"" = {x\x^X;i^{x) = w} , for i(; = 0, 1, 

where l^{x) denotes the value of the bit in the label of x dedicated to the watermark. We also define 
l\x) for j = 1, 2, . . . , m as the j-th non-watermark bit of the label of x. 

Now the question is how to choose the labeling. To maximize the noise immunity of the system, and 
since the watermark sequence is known at the receiver, we maximize the minimum distance between the 
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signal points in each of and X^. To do this, first we do a one-level set partitioning ll28l . i.e., we divide 
X into two subsets with the largest minimum distance between the points in each subset. These subsets are 
named and X^ and the watermark bit of the label is assigned accordingly. Next, by a Gray mapping 
|[29l of the signals in each of X^ and X^, the non-watermark bits of the label are assigned. This process 
is illustrated for two different signal constellations in Figs. |2] and [3] The Gray mapping ensures the least 
bit error rate in each subset ||29l . 

The minimum distance of the constellation is now defined as 

f^mm = min min ||a;j — a;j||. 

Notice that by assuming signal constellations of fixed energy, going from M-PSK in the base system to 
2M -PSK in the watermarked system does not change dmin (see Fig. |2l). For the QAM, as illustrated in 
Fig. 121 (imin does change because of energy adjustments but always stays very close to that of the original 
constellation. For example in Fig. |3l dmin is reduced by only 2.4%. 
A definition which proves useful in the next sections is 

Xj{wi,dij) = {x\x € X;£„{x) = Wi,P{x) = dij} , 

where i denotes the index of both the watermark bit and the m-bit subsequences of d and dij = 
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(a) Base system 



(b) Watermarked system 



Fig. 3. Signal constellations and their labeling for the base system (16-QAM) and the watermarked system (32-AM/PM). The 
leftmost bit in the label of the watermarked system corresponds to the watermark bit. Both constellation have unit average energy. 
Thus, dmin ~ 2 / \/T0 — 0.633 for the base system and dmin ~ 4/\/42 = 0.617 for the watermarked system. 



denotes the j-th bit of the i-th subsequence. Considering a watermark sequence of length N and an encoded 
data sequence of length inN, then i = 1,2, . . . , N. Thus, Xj{u, v) refers to the subset of X where the 
watermark bit Wi is equal to u and the j-th data bit in the i-th subsequence, i.e., j, is equal to v. The 
size of this subset is M/2. 

B. Watermark decoder 

The goal of the watermark decoder is to produce LLRs for the outer decoder given w_ and the received 
sequence y. As in |[T9l . by ignoring the correlations in d, we can use an HMM to model the received 
sequence and then use the forward-backward algorithm ll26l to calculate posterior probabilities or LLRs for 
the outer decoder. Notice that due to the nature of the channel which introduces insertions and deletions, 
there will be a synchronization drift between x and y. The synchronization drift at position i, i.e., ti is 
defined as the (number of insertions) — (number of deletions) occurred in the signal stream until the ith 
symbol, ready for transmissioriQ The drifts {U}^^^, form the hidden states of the HMM. Each 



^This means that if Xi-i is not deleted by the channel it is received as 
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state ti takes values from 

T = {..., -2,-1,0,1,2,...}. (1) 

Thus, ti performs a random walk on T whose mean and variance depend on pi and p^. To reduce the 
decoding complexity, as in |[T9l . we limit the drift to \ti\ < tmax where tmax is usually chosen large 
enough such that it accommodates all likely drifts with high probability. For example, when pi = pd> ^max 



is chosen several times larger than \/NpdJ(X^^>d) which represents the standard deviation of the drifts 
over a block of size N. 

To further characterize the HMM ||26l . we need the state transition probabilities, i.e., Pab = -P(ii+i = 
b\ti = a). Each symbol Xi entering the channel can produce any number of symbols between and / + 1 
at the channel output. As a result, if ti = a, then tj+i G {a — 1, . . . , a + /}. Notice that the transition 
from ti = a to tj+i = b can occur in two ways. One is when Xi is deleted by the channel and (6 — a + 1) 
symbols are inserted by the channel. The other one is when Xi is transmitted and {b — a) symbols are 
inserted by the channel. In either case, (6 — a + 1) symbols are produced at the channel output. As a result, 
the state transition probabilities are given by 

Pd b = a — 1 

aiPiPd +Pt b = a 

Pab = { ai{p\-^^^pA + p\-^Pt) a<b<a + I (2) 

aip{pt b = a + I 

otherwise. 

where a/ = 1/(1 — pf) is a constant normalizing the effects of maximum insertion length I and ensuring 
that the sum of probabilities is 1. 

We also need to calculate the conditional probability of producing the observation sequence y = 
{yi_^a^ ■ ■ ■ iVi+b} given the transition from ti = a to tj+i = b. As stated, this transition can occur in 
two ways. Thus, 

Q'abiy) = PiWi = = b,Wi,n) 

(i+b i+fe-1 \ 

aipt'^'pd n 7fc + «/pf->t/3i+fe n 7fc /^afe, (3) 
k=i+a k=i+a / 

where Ti denotes set of parameters of the HMM, i.e., Ti = {[Pab],T}, 7^. is the probability of receiving 
Pj. given that y/^ is an inserted symbol, and is the probability of receiving y^^^ assuming that it is 
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the result of transmitting Xi G Formally, we have 

|2 



and 



I3i+b = P{yi+b\ti = a,ti+i = b,Wi,n) = — 2_, — — )• 

Now that the HMM is defined, we use the forward-backward algorithm to calculate LLRs. By ignoring 
the correlations between the bits of d and assuming P{dij = 0) = P{dij = 1), the bit by bit LLR is 
calculated as 

''^ Pidij = l\y,w,'H) Piyjdij = l,w,n) °^ Ecc.eA-.K,!) ^^^1**'^' ^) ' 

By using the forward-backward algorithm, the posterior probabiUties are found by |[T9l . ll26l 



Piy\x,,w,n) = ^F,ia)QlMx,)B,+iib) (6) 

a,b 

where the forward quantity is defined as 

Fi{a) = Pivi,.. . = a\w,n), (7) 

the backward quantity as 

Bi{b) = P{y,^„...\U = b,w,n), (8) 



and 



Qabiyl^i) = P{y,u+i = b\ti = a,xi,n) 

i+b i+b-1 
k=i+a k=i+a 



1 ( \yi+b 



where 

A+b = P{yi^i,\ti = o, ti+i = b, Xi, n) = exp 
The forward and backward quantities are recursively computed by the forward pass 

F,{a)= F,^^{c)PMaHy^^l+c,■■■,y^-l+a), dO) 

ce{a~I,...,a+l} 

and the backward pass 

Biib) = Yl PbcQiciyi+b, ■ ■ ■,y^+c)Bi+i{c). (11) 

ce{b-l,...,b+/} 
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If the block boundaries are not known at the decoder, we can use the shding window decoding technique 
used in |[T9l . Assuming a continuous stream of transmitted blocks and received symbols, the forward- 
backward algorithm is used to infer the block boundaries. Then the decoding window is anchored at the 
most likely start of the next block and next block is decoded. Most of the results of this paper are shown 
using this sliding window decoding technique. We will briefly explain the methodology in Section IIV-AI 
For the first transmitted block, we assume that the initial drift is zero. Thus, we use 



It is also possible to insert some markers which specify the block boundaries into the transmitted 
sequence. By dedicating all the m + 1 bits in the symbols at the boundaries to markers, they can be 
easily detected at the receiver In this case, the block boundaries can be inferred by detecting the markers. 
Notice that as the block length becomes larger, recognizing the block boundaries requires less overhead 
and becomes more efficient. To see this, first define the marker rate as the number of marker symbols in 
each block divided by N. Given a fixed marker rate, the number of marker symbols is increased as N 
grows. Increasing the number of marker symbols leads to a better boundary detection at the decoder for 
a fixed pd and pi because the probability of misidentifying larger block of markers is decreased. Thus, 
for large block lengths one can assume that the block boundaries are known at the decoder and use the 
following initial conditions for the backward pass of each block: 



where c = tAr+i is the final drift at the end of the block. 

In the next stage of decoding, the LLRs, calculated by inserting Q in are passed to the outer 
decoder 

C. Outer code 

The outer code can almost be any binary error correcting code. Due to the exemplary performance of 
LDPC codes on many communication channels and their flexible structure, we choose LDPC codes in this 
paper 

At the transmitter, a binary LDPC code of rate R is used to encode the binary information sequence b 
of length niNR producing the binary coded sequence d of length mN. At the receiver, the mN bit LLRs 
of dSjl are used to recover b. 




otherwise. 



(12) 



BN{h) = P,cQUyN+b,---) 



(13) 
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IV. Error rates and fundamental limits 

In this section, we demonstrate the capabiUties of the proposed solution through examples and discus- 
sions. In particular, we evaluate the watermarked system by providing bit and word error rates (BER and 
WER), maximum achievable transmission rates, and comparing them with two benchmark systems. We 
demonstrate our results using the following two examples. 



Example 1: Consider a base system with 4-PSK modulation depicted in Fig. 2(a) which gives rise to a 



watermarked system with 8-PSK modulation. The labeling /i is chosen based on the method described in 



Section IIII-AI The constellation and labeling are depicted in Fig. 2(b) 

Example 2: In this example, we consider a 16-QAM base system and a 32-QAM watermarked system. 
Notice that different constellations can be considered for 32-ary modulation. We consider a 32-AM/PM 
constellation whose dmm is very close to the base system (they differ by only 2.4%). The constellations 
and their labelings are depicted in Fig. |3] 

A. Error rates 

First, consider Example 1. We use a (3, 6)-regular LDPC code (R = 0.5) of length 20, 024 constructed 
by the progressive edge growth (PEG) algorithm ll30l as the outer code. The LDPC code is decoded by the 
sum-product algorithm IIBTI allowing a maximum of 400 iterations. The watermark sequence w is chosen 
to be a pseudo-random binary sequence. Since m = 2, the block length is = 10, 012. The maximum 
insertion length is chosen as / = 5, the channel insertion and deletion probabilities are assumed equal, 



i.e., Pi = pd = Pid, and tmax = 5^Npid/{l -Pid)- 

A continuous stream of blocks of b, d, and x is generated and sent over the channel. A continuous 
sequence of y is then received at the decoder. We assume that block boundaries are not known at the 
receiver. Thus, y is decoded by the forward-backward algorithm using a sliding window decoding technique 
|[T9l . For the first block, we assume that the receiver knows the starting position, i.e, we use ([T2l ) to initialize 
the forward pass. For subsequent blocks, the watermark decoder is responsible to infer the boundaries and 
calculate LLRs for the outer code. This is done by first running the forward pass several multiples of tmax 
(here, six) beyond the expected position of the block boundary and initializing the backward pass from 
these last calculated forward quantities. Then the most likely drift at the end of each block is found as 
tN+i = argmaxa F/v_|_i(a)i?Ar+i(a) and is used to slide the decoding window to the most likely start of 
the next block. 

Occasionally, the watermark decoder makes errors in identifying the block boundaries. If these errors 
accumulate, synchronization is lost and successive blocks fail to be successfully decoded. To protect against 
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Fig. 4. BER and WER of the 8-PSK watermarked system employing a (3,6)-regular LDPC code of length 20, 024 versus pid at 
fixed SNRs. 



such gross synchronization loss, we use the re-synchronization technique of llT9l whose details are omitted 
in the interest of space. 

We simulate the system under different SNRs and insertion/deletion probabilities. The BER and WER 
of the system are plotted in Fig. |4] versus different values of p^^ under fixed SNRs. For example, at SNR= 
10 dB, the system is able to recover on average 1,400 symbol insertions/deletions per block of 10,012 
symbols with an average BER less than 10"^. This increases to recovering about 1, 920 insertions/deletions 
at SNR=20 dB. Fig. [5] shows the performance of the system versus SNR under fixed pij. 

We also simulate the system under a (3, 4)-regular LDPC code {R = 0.25) of length 20, 024 with the same 
parameters. The system is now capable of recovering on average 2,700 insertions/deletions per block of 
10, 012 symbols at SNR=20 dB with an average BER< 10"^. This increases to 2, 900 insertions/deletions 
using an optimized irregular LDPC code of the same rate and length with degree distributions reported in 
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Fig. 5. BER and WER of the 8-PSK watermarked system employing a (3,6)-regular LDPC code of length 20, 024 versus SNR 
at fixed values of pid. 



Table U (Code 1). We will briefly describe the LDPC optimization method in the next section. 

We are not aware of any practical coding method in the literature that can be directly and fairly compared 
to our proposed system. However, we provide comparisons with the best results of |[T9l . ijlTI . ll23l . ll24l . 
It is worth mentioning that this comparison is not completely fair as the I/D channel considered in these 
works is binary whereas in our case is non-binary. All in all, we believe that this comparison provides 
insight into what can be achieved by exploiting the extra degrees of freedom provided by non-binary 
signalling. This comparison is depicted in Fig. |6] To make the comparison as fair as possible, we have 
adjusted the block size and the rate of our 8-PSK watermarked system according to the parameters of 
codes considered in the comparison. It is evident from Fig. |6] that a significant improvement in the error 
correction performance is achieved by using non-binary signalling. There is also a significant performance 
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Pi=Pd 

Fig. 6. WER comparison of the 8-PSK watermarked system with the best results of 1191 , 1211 . 1231 . Codes D, F, and H are 
binary watermark codes from 0191 with overall rates 0.71, 0.50, 3/14, overall block lengths 4,995, 4,002, 4,662, and outer 
LDPC codes defined over GF(16), GF(16), and GF(8), respectively. Code B and E are binary marker codes from 0211 with 
overall rates 0.71 and 0.50 and overall block lengths 4, 995 and 4, 000 with binary LDPC codes as outer codes. Codes D and F 
are also decoded by the symbol-level decoding method of 1231 . All these codes are decoded on the binary I/D channel with no 
substitution eiTors or additive noise. For the non-binary I/D channel with 8-PSK signalling in Example 1, we have done sliding 
window decoding at SNR=20 dB for three different LDPC codes with variable node degree 3 and rates 0.71, 0.50, 3/14 and 
block lengths 4, 996, 4, 002, 4, 662, respectively. 

improvement compared to the method of ll24l which considers marker codes with iterative exchange of 
information between the inner and outer decoders. Marker codes concatenated with optimized irregular 
LDPC outer codes with overall rates around 0.4 and block length 5, 000 have been reported in ll24l which 
can reliably work under p^^ < 0.04. As Fig. |6] shows, a regular half-rate code with block length 4, 002 can 
do much better in our case even without iterative exchange of information. 



B. Achievable information rates 
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To obtain the capacity of the I/D channel, one is interested to calculate 11321 

where ^ is the channel input sequence of length N, y is the received sequence of random length, 
and P{x) denotes the joint distribution of the input sequence. Unfortunately, due to the presence of 
insertions/deletions, finding (fT4l ) or its bounds has proven to be extremely challenging and the capacity 
is unknown. No single letter characterization of the mutual information also exists. Most of the results in 
the literature focus on sub-optimal decoding or more constrained channel models (such as deletion-only 
channel) and provide bounds on the capacity lf33l - |[36l . Most of these bounds, however, are driven for 
binary I/D channels and binary synchronization codes and either cannot be extended to non-binary I/D 
channel or become computationally extensive such as the bounds in 1371 . 

A trellis-based approach is developed in ll38l to obtain achievable information rates for binary I/D 
channels with AWGN and inter-symbol interference under i.i.d. inputs (uniform P{x)). This approach 
which mainly uses the forward pass of the forward-backward algorithm, can be extended to i.i.d. non- 
binary inputs and thus our channel model. We will use this method to find lower bounds on the capacity 
of the channel, i.e., Q u.j,, and compare the achievable rates of our watermarked system with this lower 
bound. There also exist bounds on the perfomiance of q-ary synchronization codes [13] which we will use 
in the comparisons. 

To obtain the achievable rates of our watermarked system, we calculate the maximum average per-symbol 
mutual information. In particular, we obtain an estimate of the average mutual information between y and 
X given w- Assuming that ^ is a sequence of i.i.d. symbols, the average per-symbol mutual information 
is given by ^ Y.iLi I{xi;y\w) where 

H^uVli^l) = H{xi\wi) - H{xi\y,Wi) 



Ey 



El P{y\xi,Wi) f P{y\xi,Wi) 



L x,£X^ 



(15) 



.Y.a^,ex-. P{y\xi,wi) \Y.x,ex-^ P{y\xi,Wi 
and the conditional probabilities are given by the watermark decoder. While it is not possible to do an 
exact calculation of the expectation, it is possible to calculate it numerically by Monte Carlo simulation. 
Then, (ITSl ) can be used to find an estimate of the achievable rates of the watermarked system and a lower 
bound on the capacity of the channel. It should be noted that under large block lengths N, the variance 
of (fTSl ) under Monte Carlo simulation is usually very small. Thus, it converges to the average very fast. 
Here, our results are averaged over 100 blocks. 



18 



Using (fTSl ) and assuming known block boundaries, the achievable infomiation rates of the watermarked 
system of Example 1 is plotted versus SNR in Fig. [T] We also compare the achievable rates with those of 
the two benchmark systems. One is the base system (4-PSK) which has the same (imin> and another one 
is the system which has the same number of modulation points (8-PSK) as in the watermarked system 
but is not watermarked, i.e., all the m + 1 bits are dedicated to information bits. Both of these systems 
use Gray mapping and are decoded by the forward-backward algorithm described in Section IIII-B I with 
the exception that there is no watermark. The number of symbols per block is kept fixed at 10, 012 for all 
three systems so that the average number of symbols corrupted by insertions/deletions remains the same. 
Notice that Vc = 2.0 for the base and the watermarked system and Vc = 3.0 for the 8-PSK system with no 
watermark. 

The dashed curves in Fig. |7]correspond to the maximum achievable information rates of the three systems 
when = 0. In this case, it is clear that the watermarked and the base system achieve the same rates 
but the 8-PSK system with no watermark achieves higher rates. At = 0.01, however, the watermarked 
system performs much better than the two benchmark systems in terms of the maximum achievable rates 
(by using (ITSl) ) on the channel. This is of course not very surprising since no watermark is used in the 
benchmark systems and their only source of protection against insertions/deletions comes from the fact 
that they are decoded by the forward-backward algorithm. 

The figure also depicts Ci.u.d. under 8-PSK signalling at = 0.01. Comparing this curve with the 
results given by ([TST i shows how far the achievable rates of our watermarked system are from the maximum 
achievable rates on the channel using the same constellation under i.i.d. inputs. Although this gap is not 
small, we are not aware of any results in the literature that can approach Ci.u.d.- This gap can be made 
smaller by the method we show in Section |Vl 

We also provide a comparison with |[T9l in terms of comparing the achievable rates of these two systems 
as viewed by the outer code. In particular, we calculate the average I{dij; lij) which is a number between 
and 1 for both systems. This is done by Monte Carlo simulation, using the LLRs produced by the 
watermark decoder, i.e., using (|5). This comparison is depicted in Fig. [8] The achievable rates seen by 
the outer code from ||T9l are compared to those of the watermarked 8-PSK system at SNR=20 dB. The 
rates of |[T9l are given for three binary watermark codes with sparsifier rates of 3/7, 4/6, and 4/5 and 
are calculated assuming no substitution error on the channel which is similar to the high SNR case on our 
channel. As depicted, the achievable rates of the proposed watermarked 8-PSK system are much higher 
than those of HI. 

Given the success of LDPC codes on many channels, we expect that the information rates of Fig. [8] can 
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Fig. 7. Maximum achievable information rates (bits per channel use) versus SNR under different modulations assuming a 4-PSK 
base system (rc = 2.0). The 8-PSK watermarked system mentioned in Section IIII-AI has rc — 2.0. The maximum achievable 
rates given by ( |15t under 8-PSK modulation with no watermark (rc = 3.0), and under two 8-PSK watermarked system with 
partial watermarking (rc — 2.3 and 2.8) mentioned in Section|V]are plotted for comparison. Also, Ci.u.d. is plotted for the 8-PSK 
modulation. 



be approached with carefully designed irregular LDPC codes of large block lengths. To demonstrate this, 
we have optimized the degree distributions of irregular LDPC codes of rates 0.25, 0.50, and 0.75, and 
constructed codes of length 20, 024. The optimization process is done by the conventional numerical LDPC 
optimization methods in the literature (e.g., see |!39|). These optimization techniques usually use the pdf 
of the LLRs. On most channels, this LLR pdf can be calculated analytically. However, this cannot be done 
in our case due to nature of the channel. Thus, Monte Carlo simulation is used to find estimates of the 
LLR pdf. Given the channel parameters, this is done by simulating a large number of channel realizations, 
calculating LLRs using ([5]), and finally computing the average LLR pdf (probability mass function to be 
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Fig. 8. Maximum achievable information rates as viewed by the outer code I{d,ij;lij) for the 8-PSK watermarked system at 
high SNR are compared with the obtained rates of 1191 . The rates of 1191 are calculated assuming no substitution errors. Also, 
the maximum pid that the 8-PSK watermarked system can tolerate with BER less than 10^'' is indicated for the three optimized 
irregular LDPC codes of rates 0.25, 0.50, and 0.75 and three regular LDPC codes. 



more precise). Next, the rate of the code is maximized by optimizing its degree distributions using the 
computed LLR pdf. The optimized degree distributions are given in Table. Jl After optimizing the degree 
distributions, the parity-check matrices of the codes are constructed by the PEG algorithm ll30l . Finally, 
the codes are simulated on the channel at high SNR by assuming known block boundaries with the rest of 
parameters being the same as in Example 1. Fig. [8] shows the maximum under which the constructed 
irregular LDPC codes perform with BER less than 10^^. It is seen that these practically achievable rates 
are not far from the maximum achievable rates given by I{di j] lij). Also depicted in Fig. [8] are the results 
for three regular LDPC codes of the same length, i.e., (3, 4)-regular, (3, 6)-regular, and (3, 12)-regular 
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TABLE I 

Variable and check node degree distributions for the optimized irregular LDPC codes; All results 
achieved assuming maximum variable node degree of 30 





Code 1 


Code 2 


Code 3 


A2 


0.2793 


0.1920 


0.2562 


A3 


0.2648 


0.2480 


0.3334 


A4 




0.0064 


0.0010 


As 






0.0022 


Ae 


0.0173 


0.0171 


0.3621 


A7 


0.0575 


0.0709 


0.0434 


As 


0.0938 


0.1223 


0.0017 


A9 


0.0279 


0.0278 




Aio 


0.0528 


0.0302 




A18 




0.0413 




A26 


0.0494 


0.0302 




A27 


0.0126 


0.0137 




A28 


0.0179 


0.0199 




A29 


0.0303 


0.0358 




A30 


0.0964 


0.1507 




P5 


1.0000 






P9 




1.0000 




Pis 






1.0000 


Rate 


0.25 


0.50 


0.75 



LDPC codes with rates 0.25, 0.50, and 0.75, respectively. 

Now, consider Example 2. Using the same method, the maximum achievable information rates of the 
watermarked system (by using (ITST l) are compared to those of the two benchmark systems (Gray labeled 
16-QAM in Fig. |3(a)| and quasi Gray 32-QAM) and Ci.u.d. in Fig. |9l The block size is again kept fixed at 
10, 012 symbols. The same discussion as in Example 1 applies to this case as well. 

We also compare the maximum achievable information rates with the bounds available for (7-ary synchro- 
nization codes. We use the asymptotic bounds for q-ary codes of iflBl to compare with our scheme. These 
bounds are achieved by considering the Levenshtein distance H between g-ary codewords and enumerating 
the maximum size of codes capable of correcting insertions/deletions with zero error probabilities. Since 
these bounds do not consider substitution errors or additive noise, we compare them with the achievable 
rates of our scheme in the high SNR region. 

Fig. [To] compares the upper and lower bounds of g-ary codes for g = 8, 32, with the achievable 
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Fig. 9. Maximum achievable infomadon rates (bits per channel use) versus SNR under different modulations assuming a 16- 
QAM base system (rc = 4.0). The 32-AM/PM watermarked system mentioned in Section UlI-AI has Tc — 4.0. The maximum 
achievable rates given by dlS) under quasi-Gray 32-QAM modulation with no watermark (rc = 5.0), and under two 32-AM/PM 
watermarked system with partial watermarking (rc = 4.3 and 4.8) mentioned in Section |V] are plotted for comparison. Also, 
Ci.u.d. is plotted for the 32-QAM modulation. 



infomiation rates of our 8-PSK and 32-AM/PM schemes using dTSl) . and also Cj.u.d.- Notice that Cj.u.d. and 
our achievable information rates in some regions exceed the upper bound of [13|. This is due to the fact 
that the upper bounds are computed assuming zero error probabilities whereas Ci.u.d. and our achievable 
rates given by ([T5] ) are computed assuming asymptotically vanishing error probabilities and thus computed 
in different scenarios. As seen on the figure, the achievable rates of our watermarked system is below 
Ci.u.d. and in some regions below the q-ary upper bound. Nevertheless, no code exists in the literature 
which can approach these limits. At small pid, the q-ary codes can theoretically achieve higher information 
rates than our scheme. This suggests that it is not efficient to dedicate one whole bit to the watermark 
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Fig. 10. Maximum achievable information rates of the watermarked systems at high SNR are compared to the upper and lower 
bounds of g-ary insertion/deletion correcting codes 0131 and the achievable rates Ci.u.d.- These rates are plotted for the 8-PSK 
and 32-AM/PM watermarked systems in two cases each. First, using binary watermark and assigning one bit to the watermark 
in each symbol (r-c = 2.0 and rc — 4.0 for 8-PSK and 32-AM/PM, respectively). Second, the achievable rates are maximized by 
optimizing gw and rc in each point. 



when the number of synchronization errors is small. We will show in the next section how the information 
rates can be increased. 

V. Increasing the achievable information rates 

In this section, we show how the maximum achievable rates of the watermarked system can be improved. 

We defined rc to be the average number of bits assigned to the information bits (more precisely coded 
bits) per each transmitted symbol. Until now, we considered cases where one bit was assigned to the binary 
watermark in each of the transmitted symbols. For example, for the 4-PSK base system and the 8-PSK 
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watermarked system discussed in Example 1, rc = 2.0 bits. It is also possible to embed watermark bits 
into only some of the symbols but not all of them. For 8-PSK, this means 2.0 < Vc < 3.0. We use Gray 
mapping for those symbols which are not watermarked. Also, we scatter those symbols which contain 
watermark uniformly in the transmitted block. 

It is evident that there is a trade-off between rc and the system ability to recover synchronization errors. 
Increasing Vc potentially lets more information to pass through the channel but at the same time increases 
the system vulnerability to synchronization errors since less bits are assigned to the watermark. As a 
result, for a fixed signal set, there exists an optimum rc for each and SNR which provides the largest 
transmission rate on the channel. 

For example consider the system of Example 1. At high SNR and pid = 0.01, the maximum achievable 
rate is 1.945 bits per channel use when rc = 2.0 bits which increases to 2.528 bits per channel use when 
Tc = 2.8 bits. This implies that dedicating one bit in every symbol to the watermark, i.e., rc = 2.0 is 
wasteful in this case. A better protection is provided against synchronization errors by assigning one bit 
to the watermark in only 20% of the symbols. As examples. Fig. |7] also shows the maximum achievable 
rates of the 8-PSK system under rc = 2.3 and rc = 2.8. For pid = 0.01, when SNR< 6.44 dB the 
system with rc = 2.0 achieves higher rates compared to those of systems with rc = 2.3 and rc = 2.8. 
When 6.44 <SNR< 9.31, rc = 2.3 provides the largest rates and when SNR>9.31 dB, rc = 2.8 provides 
the largest rates compared to the other two systems. Fig. |9] depicts the comparison for the 16-QAM and 
32-AM/PM systems. 

Until now, we have only considered binary watermark sequences. When the number of synchronization 
errors is large, a binary watermark may not be very helpful in localizing these errors. Increasing the 
alphabet size of the watermark increases the system ability to combat synchronization errors. Increasing 
(7w> however decreases rc, i.e., less number of bits are available in each symbol for information bits. As 
a result, there is a trade-off between the g„, rc, and the maximum achievable information rate on the 
channel. 

Fig. [To] depicts the maximum achievable rates that 8-PSK and 32-AM/PM watermarked systems can 
achieve by finding the optimum rc and q„ at each point. It is evident that the maximum achievable rates 
can be increased significantly by this strategy. This is especially beneficial when pi^ is small where the 
achievable rates gets closer to the lower bound on g-ary codes. 
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VI. Complexity and the watermark sequence 

A. Decoding complexity 

The complexity of the forward-backward algorithm determines the complexity of watermark decoding. 
This complexity scales as O ((1 + 2tina.x)IMN), where l + 2tinax is the number of states in the HMM and 
N is equal to the number of symbols on which the forward-backward algorithm is performed. It should 
be noted that it is possible to reduce this complexity using arguments similar to those of |[T9l . 

B. Watermark sequence 

The watermark sequences used in this paper are pseudo-random sequences. Our experiments confirm 
that these sequences perform well under different insertion/deletion rates. Periodic sequences with small 
periods are usually not good choices. First, they are vulnerable to successive insertios/deletions. As a 
simple example, if the watermark is a periodic sequence with period 4, then the decoder cannot detect 4 
successive deletions. Furthermore, certain patterns of insertions/deletions can fool the decoder such that it 
fails to detect them. Randomness lowers the probability of false detecting or missing insertions/deletions 
by the decoder. 

Among other factors which affect the decoding performance is the number of successive identical symbols 
(runs) in the watermark. One advantage of having runs is that it provides the ability to detect successive 
deletions. The larger the run-length, the larger successive deletions that can be detected. Nevertheless, 
larger runs lead to a worse localization of insertions/deletions as the decoder is not able to detect where 
exactly insertions/deletions have occurred. This gives rise to less reliable LLRs in the vicinity of inser- 
tions/deletions. Thus, there should be a balance between large and small runs in the watermark sequence. 
Pseudo-random sequences usually have this property. 

Among candidates for the watermark are the run-length limited (RLL) sequences. RLL sequences with 
small maximum run-lengths (e.g., around 3 or 4) are good choices particularly when p■■^^ is very small 
since the probability of having successive insertions/deletions is small. The performance gain over pseudo- 
random watermarks is only notable at small p^^ where high-rate outer codes are used. 

The presence of additive noise can also affect the choice of watermark. All in all, it is possible that 
sequences with structure could offer better perfomiance than the above-mentioned sequences. This can be 
the subject of further investigation. 
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VII. Conclusion 

In this paper, we proposed a concatenated coding scheme for reUable communication over non-binary 
I/D channels where symbols were randomly deleted from or inserted in the transmitted sequence and all 
symbols were corrupted by additive white Gaussian noise. First, we provided redundancy by expanding 
the symbol set while maintaining almost the same minimum distance. Then, we allocated part of the bits 
associated with each symbol to watermark bits. The watermark sequence, known at the receiver, was then 
used by the forward-backward algorithm to provide soft information for the outer code. Finally, the received 
sequence was decoded by the outer code. 

We evaluated the performance of the watermarked system and through numerical examples we showed 
that significant amount of insertions and deletions could be corrected by the proposed method. The 
maximum information rates achievable by this method on the I/D channel were provided and compared 
with existing results and the available bounds on q-ary synchronization codes. Practical codes were also 
designed that could approach these information rates. 
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